The present invention relates to computer networks. More particularly, the present invention relates to network security systems for detecting and protecting against security breaches (both internal and external), network failures, and other types of data-compromising events.
A significant problem in the field of computer networks has been the inability to adequately protect private Internet-connected computer networks from security attacks. This problem commonly arises, for example when a company interconnects its internal network (typically a local area network) with the Internet to allow company employees to more easily communicate with outside entities. The benefits of connecting the internal network to the Internet are often significant, including, for example, enabling the company to inexpensively disseminate product information and provide online customer support to potential and existing customers.
As many companies have discovered, however, connecting the internal network to the Internet can have devastating consequences in the absence of an adequate security mechanism. A break-in by a hacker, for example, will often result in the deletion of important data or software files, the introduction of a virus to the network, and/or the public dissemination of confidential information. Less overt break-inns may involve the secret misappropriation of company trade secrets, or the covert manipulation of company data files. Even an innocent act by a company employee, such as the downloading of a virus-ridden file from a Web site, can have devastating effects.
One type of security system which provides limited protection against intrusions is a network firewall system (xe2x80x9cfirewallxe2x80x9d). A firewall is a computer system that restricts the flow of traffic between two networks based on a pre-programmed access control policy. One type of firewall, commonly referred to as a network-level firewall, filters the traffic at the packet level based on the source and destination IP (Internet Protocol) addresses and IP ports of the packets. Another type of firewall masks the internal addresses of the private network, making these addresses appear as firewall addresses. Other firewalls implement elaborate logon and user authentication schemes.
One problem with existing firewall systems is that they are generally only effective at protecting against known types of security attacks. Once a user determines how to circumvent the firewall""s access control policy, the firewall offers little or no protection. Although some firewalls generate audit trails of intrusion attempts, these audit trails typically do not reflect the attacks that are actually successful, and are therefore of little value to identifying either the intruder or the method of intrusion. Moreover, even when a successful security attack is recorded within the audit trail, the audit trail will rarely contain the information needed to determine the extent of the damage, let alone restore the network to its pre-intrusion state.
Another problem with existing firewall systems is that they perform little or no virus checking on incoming file transfers, and, even when virus checking is performed, detect only known types of viruses. Yet another problem with firewalls, and with other types of network security systems, is that they do not provide an adequate mechanism for detecting and tracking malicious acts that are performed xe2x80x9con-sitexe2x80x9d by company employees.
The present invention seeks to overcome these and other deficiencies in existing network security systems.
The present invention provides a network security and analysis system which includes a variety of features for automatically and interactively monitoring and analyzing traffic on a LAN (local area network), WAN (wide area network), or other type of computer network. In the preferred embodiment, the system is implemented using a general-purpose computer which passively captures and monitors the bi-directional traffic appearing on a network or network segment. The system includes software modules for analyzing the passively-captured traffic in both automatic and interactive (off-line) analysis modes. The system can, but need not, be used in conjunction with a network firewall.
In accordance with one aspect of the invention, the system continuously captures all valid data-link-level packets, and routes this traffic (together with date/time stamps) to a high-capacity, non-volatile data recorder to generate a low-level archival recording. The storage device may, for example, be a high-speed magnetic tape drive. In the preferred embodiment, the system captures and records the packets passively (i.e., in a receive-only mode). Thus, the system does not add latency to the normal flow of traffic on the network, and the system""s presence is virtually undetectable by other entities of the network.
In one embodiment, the data-link-level traffic is captured using an off-the-shelf network interface card that connects to the network at a network monitoring point. Through appropriate software, the card is configured to continuously return all bi-directional packet data to the general-purpose host computer at the lowest data interface level supported by the card. In another embodiment, a custom or a modified network interface card is used which enables the system to additionally capture and record packet fragments, collision events, synchronization sequences, and other types of transmission sequences that are typically not accessible when an unmodified off-the-shelf card is used. In yet another embodiment, an analog recorder is used to record the electrical signal present on the network.
An important benefit of the recording process is that the traffic is captured and recorded at the data-link level, which is the lowest-level protocol at which data is transferred as packets. Because the traffic data is recorded at this level (or a lower level), the recordings can subsequently be used to fully-reconstruct and evaluate virtually any type of network transaction that takes place on the network, regardless of the protocol level at which these transactions occur. The archival recordings can therefore be used to perform a wide range of network analysis and restoration tasks, including, for example, restoring lost data files, checking previously-downloaded files for newly-discovered viruses, and performing low-level analyses of network break-ins. The archival recordings are also useful to a variety of network troubleshooting tasks, particularly when a custom or modified network interface card is used that provides access to invalid packet transmissions.
Another benefit of the recording process is that it proceeds continuously, as opposed to being contingent upon the detection of pre-programmed network events. Thus, unlike the event-triggered audit trails generated by many firewall systems, the archival recordings can be used to detect and analyze break-ins and other network anomalies that are not detected at the time of occurrence.
In accordance with another aspect of the invention, the packet stream is optionally encrypted by the system prior to being written to the archival storage medium. Any of a variety of known data encryption methods can be used for this purpose. One benefit of encrypting the packet stream is that it significantly hinders the covert manipulation of the archival recordings by unauthorized users (intruders, company employees, etc.). The archival recordings consequently serve as highly reliable evidence of the events that have taken place on the network, and can be used, for example, as legal proof of user misconduct. Another benefit of encrypting the packet stream is that it protects the privacy of the communications recorded within the archival recordings, and provides for the authentication of the recorded data.
In accordance with another aspect of the invention, the system includes various software and hardware components for allowing an authorized user to analyze the archived traffic data in an xe2x80x9coff-linexe2x80x9d mode. These components include a data playback unit for replaying archival recordings, a set of traffic analysis databases for storing the archived traffic data during analysis, and a set of post-capture analysis applications. In other embodiments, the off-line analysis functionality is allocated to a separate general-purpose computer.
In operation, the data playback unit is initially used to stage the archived traffic data (or selected portions thereof) to the traffic analysis databases. During this process, user-configurable filters can optionally be invoked to select the types of packets that are loaded into the databases. The analysis applications can then be used to view, analyze and process the traffic data. These applications include functionality for performing such actions as displaying user-specified types of network events, conducting pattern searches of selected packet data, reconstructing transaction sequences, and identifying pre-defined network problems. A lost data recovery application allows the user to reconstruct traffic sequences for subsequent playback onto the network, and can be used, for example, to restore lost data files following a break-in, or to replay an error-causing traffic sequence. Files can also be recovered directly off the recorded data for some protocols, including HTTP (HyperText Transport Protocol) and FTP (File Transfer Protocol).
Another aspect of the invention is a method for recording non-network transactions within the archival recordings. These non-network transactions are events which take place on user computers of the network without a corresponding transmission of data over the network. Examples of such transactions include e-mail messages sent and received using a locally-attached modem, and deletions of locally-stored data files. To enable such transactions to be monitored, a special transaction replication application is provided which runs transparently on one or more user computers. When a non-network transaction of a pre-specified type occurs on a user computer, the replication application encapsulates the transaction (or a representation thereof) using a pre-defined message format, and transmits the encapsulated transaction on the network. This causes the encapsulated transaction to be captured within the archival recording, and allows the transaction to be monitored.
In accordance with another aspect of the invention, the system automatically monitors the passively-captured packet stream in near-real-time to check for a variety of pre-programmed anomalies. In the preferred embodiment, this is accomplished in-part via software which continuously routes at least some of the passively-captured traffic data to a cyclic data recorder. The cyclic data recorder is preferably separate from the data recorder used to generate archival recordings, and is used to temporarily store the traffic data for automated post-capture analysis. When this feature is enabled, a real-time monitoring application reads the traffic data from the cyclic recorder on a first-in-first-out basis and checks for pre-programmed anomalies. Both visual and audible alarms are provided for alerting authorized users when an anomaly is detected. Because the traffic data is analyzed only after being passively captured, thorough analyses (extensive virus checks, reconstruction of transaction sequences, etc.) can be performed without any interruption to the normal flow of data on the network.
In accordance with another aspect of the invention, a configuration option is provided in which the system passively monitors and records the data-link-level traffic appearing on both sides of a commercially-available network firewall. A real-time monitoring application in-turn compares the pre-firewall and post-firewall recordings to identify specific transactions that are rejected by the firewall. In one embodiment, the results of this traffic comparison are automatically compared to a rejection log maintained by the firewall to check for inconsistencies.